.\" SPDX-License-Identifier: BSD-2-Clause
.\" Copyright (C) 2021 Intel Corporation.
.\"
.TH "MEMTIER" 7 "2021-03-01" "Intel Corporation" "MEMTIER" \" -*- nroff -*-
.SH "NAME"
libmemtier.so - interposer library which enables the memkind memory tiering

.SH "SYNOPSIS"
In order to use the memkind memory tiering, pass the following environment variables along with the command:
.IP
.B LD_PRELOAD=libmemtier.so MEMKIND_MEM_TIERS="..."
command {arguments ...}

.SH "DESCRIPTION"
This library enables the memkind memory tiering mechanism. With this funcionality, allocations will be split between
different types of memory automatically. The library allows making allocations with the usage of multiple kinds keeping
a specified ratio between them. This ratio determines how much of the total allocated memory should be allocated with
each kind. Both
.BR LD_PRELOAD
and
.BR MEMKIND_MEM_TIERS
environment variables are mandatory for enabling the memkind memory tiering.

.SH "ENVIRONMENT"
.TP
.B MEMKIND_MEM_TIERS
A semicolon-separated list of tier configurations ended with a policy parameter. Each of the tier configurations is
a comma-separated list of tier parameters in the format:
.BR "param_name:value,(...)"
For the list of available tier parameters, please see the
.BR "TIER PARAMETERS"
section below.
.TP
.B MEMKIND_MEM_THRESHOLDS
Semicolon-separated list of threshold configurations. Each configuration consists of a comma-separated list of threshold
parameters in the same format as
.B MEMKIND_MEM_TIERS
, where the
.B param_name
is one of the
.I INIT_VAL, MIN_VAL
and
.I MAX_VAL.
Each parameter can be set only once. When any of the parameters are missing, they will be set to a default value.
If threshold configuration is missing, threshold with default configuration will be used. If any of the parameters is set
more than once, the application will be aborted. Order of provided configurations is important - configuration of Nth threshold
defines threshold between Nth and (N+1)th tier. Note that for N+1 tiers there should be at most N thresholds defined.
Set this variable only when
.BR DYNAMIC_THRESHOLD
policy is used. For more details about
.B param_name
see the
.BR "THRESHOLD PARAMETERS"
section below. See also
.BR EXAMPLES
section for the usage of
.BR MEMKIND_MEM_THRESHOLDS
environment variable.

.SH "TIER PARAMETERS"
.TP
.B KIND
(required) - kind of memory used in memory tier. Allowed kind names are
.I 'DRAM', 'KMEM_DAX'
and
.I 'FS_DAX'.
There can be multiple different kind names provided in the
.B MEMKIND_MEM_TIERS
environment variable - one in each tier configuration. Additionally, in this variable more than one
.I 'FS_DAX'
kind name is allowed.
More information about available kinds can be found in the
.B memkind(3)
manual, where the
.I 'DRAM'
kind name allocates memory with the usage of the
.B MEMKIND_DEFAULT
kind, the
.I 'KMEM_DAX'
with the
.B MEMKIND_DAX_KMEM
kind and the
.I 'FS_DAX'
with a file-backed kind of memory created with the
.B memkind_create_pmem()
function.
.TP
.B PATH
(required only for the
.I FS_DAX
kind) - the path to the location where pmem file will be created. The path has to exist. For other kinds, if set, will
cause an error.
.TP
.B PMEM_SIZE_LIMIT
(optional, only for the FS_DAX kind) - if set, it limits the size of pmem file and the maximum size of total
allocations from persistent memory. By default, no limit is introduced. Pass this option only with
.I FS_DAX
kind. For other kinds, if set, will cause an error.
The accepted formats are: 1, 1K, 1M, 1G. See the
.B memkind(3)
manual for information about limitations to this value which are the same as for the
.B max_size
value of the
.B memkind_create_pmem()
function.
.TP
.B RATIO
(required) - the part of the ratio tied to the given kind. It's an
.I unsigned
type in a range from 1 to
.I UINT_MAX.
See
.B EXAMPLES
section.
.TP
.B POLICY
(required, only one in the whole environment variable) - determines the algorithm used to distribute allocations between
provided memory kinds. This parameter has to be the last parameter in
.BR MEMKIND_MEM_TIERS
configuration string. Currently only
.I STATIC_RATIO
and
.I DYNAMIC_THRESHOLD
policies are valid. See the
.BR POLICIES
section.
.PP
.br
.BR NOTE:
The application will fail when provided environment variable string is not in the correct format.

.SH "POLICIES"
.TP
.B STATIC_RATIO
All allocations are made in such a way that the ratio between memory tiers is constant. No threshold is used.
.TP
.B DYNAMIC_THRESHOLD
The ratio between memory tiers is kept with a help of a threshold between kinds which value changes in time.
Minimum two tiers are required for this policy, otherwise the application will be aborted. The threshold value can change
in time to keep the desired ratio between tiers, but it will not be lesser than
.I MIN_VAL
and it will not be greater than
.I MAX_VAL.
For every allocation, if its size is greater than or equal to
.I INIT_VAL,
it will come from the (N+1)th tier.
.IP
Default values for a threshold between first two tiers in the
.B MEMKIND_MEM_TIERS
environment variable are:
.IP
.B INIT_VAL = 1024, MIN_VAL = 513, MAX_VAL = 1536.
.IP
If there are more tiers defined, each next undefined threshold will have all parameters increased by 1024,
so the next undefined threshold between the next two tiers will have:
.IP
.B INIT_VAL = 2048, MIN_VAL = 1537, MAX_VAL = 2560.

.SH "THRESHOLD PARAMETERS"
.TP
.B INIT_VAL
(optional) - the initial value of the threshold between two adjacent Nth and (N+1)th tiers. It must be greater than or equal to
.I MIN_VAL
and less than or equal to
.I MAX_VAL.
.TP
.B MIN_VAL
(optional) - the minimum value of the threshold.
.TP
.B MAX_VAL
(optional) - the maximum value of the threshold.
.PP
.BR NOTE:
Because setting the above parameters is optional, they will be set to default values in case they are not defined.

.SH "DRAM FALLBACK POLICY"
With the usage of both
.I DRAM
and
.I KMEM_DAX
tiers, if there is not enough memory to satisfy the
.I DRAM
tier memory allocation request, the allocation will fall back to PMEM memory.
.PP
With the usage of both
.I DRAM
and
.I FS_DAX
tiers, if there is not enough memory to satisfy the
.I DRAM
tier memory allocation request, the allocation will fail.
.PP
If there is not enough memory to satisfy the
.I FS_DAX
or
.I KMEM_DAX
tier memory allocation request, the allocation will fail.

.SH "EXAMPLES"
.br
The following example will run
.B ls
with the memkind memory tiering library. Make sure that paths to both
.B libmemtier.so
and
.B libmemkind.so
are included in
.B LD_LIBRARY_PATH.
During the application run, 20% of the allocated memory will come from PMEM memory and 80% will come from DRAM (
.I FS_DAX:DRAM
ratio is 1:4):
.IP
.B LD_PRELOAD=libmemtier.so MEMKIND_MEM_TIERS="KIND:FS_DAX,PATH:/mnt/pmem0,PMEM_SIZE_LIMIT:10G,RATIO:1;KIND:DRAM,RATIO:4;POLICY:STATIC_RATIO" /bin/ls -l
.PP
The example value of
.BR MEMKIND_MEM_TIERS
environment variable where all allocations will come from PMEM memory with filesystem created with the path /mnt/pmem0
(PMEM file size is limited only by the specified filesystem):
.IP
.B LD_PRELOAD=libmemtier.so MEMKIND_MEM_TIERS="KIND:FS_DAX,PATH:/mnt/pmem0,RATIO:1;POLICY:STATIC_RATIO"
.PP
The example value of
.BR MEMKIND_MEM_THRESHOLDS
environment variable. With
.I INIT_VAL=64,
on the application start all allocations lower than 64 bytes threshold will come from DRAM and equal to or greater than
this value will come from PMEM memory NUMA nodes. The threshold value changes during the runtime in order to maintain the ratio.
.I MIN_VAL=1
and
.I MAX_VAL=10000
set the lower and upper limits of the threshold value. Note that the
.I DYNAMIC_THRESHOLD
policy has to be set in
.BR MEMKIND_MEM_TIERS
environment variable:
.IP
.B LD_PRELOAD=libmemtier.so
.B MEMKIND_MEM_TIERS="KIND:DRAM,RATIO:1;KIND:KMEM_DAX,RATIO:4;POLICY:DYNAMIC_THRESHOLD"
.B MEMKIND_MEM_THRESHOLDS="INIT_VAL=64,MIN_VAL=1,MAX_VAL=10000"

.SH "NOTES"
.B libmemtier
works for applications that do not statically link a
.B malloc
implementation.

.SH "COPYRIGHT"
Copyright (C) 2021 Intel Corporation. All rights reserved.

.SH "SEE ALSO"
.BR memkind(3),
.BR malloc(3)
